Cross Validation

• Cross-validation (CV) is a way to choose optimal hyperparameters using the training data
• Split the training data into subsets, e.g., A, B, C, D, E
• Define a finite set of hyperparemeter combinations (a grid) to choose from
  • Example: {“max_depth”: [3, 4], “learning_rate”: [0.05, 0.1]}
  • Example: {“hidden_layer_sizes:[[4, 2], [8, 4, 2], [16, 8, 4]]}

• Use one of the subsets (e.g., A) as the validation set
• Train with each of the hyperparameter combinations on the union of the remaining subsets (e.g., B \(\cup\) C \(\cup\) D \(\cup\) E)
• Compute the trained model scores on A
• Repeat with B as the validation set, etc.
• For each hyperparameter combination, end up with as many validation scores as there are subsets

• Average the validation scores to get a single score for each hyperparameter combination
• Choose the hyperparameters with the highest average score
• All of this together is “search over the grid using cross-validation to find the best hyperparameters”
• It is implemented by scikit-learn’s GridSearchCV function

Example

• Same data as in 3a-trees
  • agr, bm, idiovol, mom12m, roeq
  • data = 2021-12 (training data)
• Quantile transform features and ret

Cross validate gradient boosting

from sklearn.ensemble import GradientBoostingRegressor
from sklearn.model_selection import GridSearchCV